Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix tiktok download #20

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix tiktok download #20

wants to merge 2 commits into from

Conversation

GreyAlien502
Copy link

Currently, tiktok downloads fail with status 403.
This request fixes that by downloading the page twice before trying the video.
This seems like a bug on their end since i noticed the same problem in the webapp:
the request for the video file fails with 403 when opening a video in a fresh incognito session, but it works when i reload.

merval
merval reviewed Oct 26, 2020
View reviewed changes
Copy link

@merval merval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the approach, but I don't think you need to assign the webpage variable twice. simply calling self._download_webpage does the trick

youtube_dlc/extractor/tiktok.py Outdated Show resolved Hide resolved
@blackjack4494
Copy link
Owner

Unfortunately there is no retry option for downloading webpages or info files like in json.
However the mentioned approach here is poorly implemented.
I don't have any problems even trying multiple times. You could at most implement a solution that checks the http status and retries again after a certain timeout.
Currently you force even users without problems to download the page twice.
Also have you tried something like --force-ipv4 or --force-generic?

@merval
Copy link

merval commented Oct 27, 2020

@GreyAlien502 We should look at implementing what @blackjack4494 is referring to. Check the page status code, and call a second time if we get 503 back.

@GreyAlien502
Copy link
Author

@blackjack4494, We aren't retrying anything here,
especially not the webpage/info file download, which never fails,
The second call is preemptive, to avoid getting a 403 later, when we get the video.

I don't have any problem

Oh, i wasn't aware that the current code was working for anyone (without providing cookies).
It always fails for me.

have you tried something like --force-ipv4 or --force-generic

I just tried --force-generic, and it sometimes works for me.
Since it makes multiple requests for each resource it downloads,
it's not too surprising that it would.

@GreyAlien502
Copy link
Author

GreyAlien502 commented Oct 27, 2020 •
edited
Loading

@merval, an InfoExtractor will not know that there's a problem
since the error doesn't occur during the information extraction, right?

@GreyAlien502 GreyAlien502 mentioned this pull request Oct 27, 2020
Closed
5 tasks
blackjack4494
blackjack4494 reviewed Nov 4, 2020
View reviewed changes
youtube_dlc/extractor/tiktok.py
Comment on lines +137 to 138
self._download_webpage(url, video_id)
webpage = self._download_webpage(url, video_id, note='Downloading video webpage')
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change this so that it won't download the webpage twice if not necessary. You can check the result and implement some try again but only once if response isn't satisfying.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If anyone can provide a way of determining if the response is satisfying, i can add such a check.
I don't see any meaningful difference between the first and second responses i receive.
Is there a specific reason why you think such a check would be possible?

@shiwei87tan
Copy link

tiktok download is down again.

U:\youtube-dlc -i -o "%(id)s.mp4" -v https://www.tiktok.com/@tiktok_stage/video/6885152542095887617
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-i', '-o', '%(id)s.mp4', '-v', 'https://www.tiktok.com/@tiktok_stage/video/6885152542095887617']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.07
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg git-2020-08-07-fab00b0, ffprobe git-2020-08-07-fab00b0
[debug] Proxy map: {}
[TikTok] 6885152542095887617: Downloading video webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://v16-web.tiktok.com/video/tos/alisg/tos-alisg-pve-0037c001/9b3784d6dec84c75b0233ca0d90572c5/?a=1988&br=1272&bt=636&cr=0&cs=0&cv=1&dr=3&ds=3&er=&expire=1604852024&l=202011081013320101151511041701C791&lr=tiktok&mime_type=video_mp4&policy=2&qs=0&rc=M2x3OWhzaTtpeDMzaTgzM0ApZjhoZzczPGU7NzdnNGc5O2dtLmhoaGtxZzNfLS02LzRzc2JeNGM2MzUxLjAzXzVjMDM6Yw%3D%3D&signature=7773bd3019bd845e9f8ee935e9ef478b&tk=tt_webid_v2&vl=&vr='
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
File "youtube_dlc\YoutubeDL.py", line 2011, in process_info
File "youtube_dlc\YoutubeDL.py", line 1866, in dl
File "youtube_dlc\downloader\common.py", line 373, in download
File "youtube_dlc\downloader\http.py", line 349, in real_download
File "youtube_dlc\downloader\http.py", line 114, in establish_connection
File "youtube_dlc\downloader\http.py", line 110, in establish_connection
File "youtube_dlc\YoutubeDL.py", line 2316, in urlopen
File "urllib\request.py", line 523, in open
File "urllib\request.py", line 632, in http_response
File "urllib\request.py", line 561, in error
File "urllib\request.py", line 494, in _call_chain
File "urllib\request.py", line 641, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

@GreyAlien502
Copy link
Author

btw, this was fixed in ytdl-org/youtube-dl by adding the extra call to tiktok in the _real_initialize method.

@pukkandan pukkandan mentioned this pull request Jan 7, 2021
Merged
pukkandan referenced this pull request in yt-dlp/yt-dlp Jan 7, 2021
@pukkandan
Tiktok fix #8 (blackjack4494#20)
9d88274 
Authored by GreyAlien502
@tanliang2
Copy link

how do you have fixed the bug downloading tiktok resposnse with 403? i have the same probem to download the url
"https://v16-web.tiktok.com/video/tos/alisg/tos-alisg-pve-0037c001/5d1561cb4b5c4e94b68c2d8aefc2526b/?a=1988&br=4898&bt=2449&cd=0%7C0%7C1&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1609504918&l=2021010106413301011515316709B85BC4&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=anU0bHY6bG46eDMzNTczM0ApNTRlaTczNmU8N2lkZmc8PGdpYmkyMm9rXmJfLS0vMTRzcy40MTAwNF82YmFjMy41LzM6Yw%3D%3D&signature=8028848b03a13b99813cb39eb52e5b11&tk=tt_webid_v2&vl=&vr="

@pukkandan
Copy link
Contributor

pukkandan commented Jan 14, 2021 •
edited
Loading

@tanliang1 Provide the full verbose output (with -v)

Also, you might want to try it on the master of yt-dlp. I have applied a tiny patch on top of this PR

pukkandan added a commit to yt-dlp/yt-dlp that referenced this pull request Jan 14, 2021
@pukkandan
[tiktok] Fix for when share_info is empty
2e8d262 
(Related: blackjack4494/yt-dlc#20)
siikamiika pushed a commit to siikamiika/yt-dlc that referenced this pull request Feb 15, 2021
@pukkandan
[TrovoLive] Add extractor (partially fix blackjack4494#20)
5c61051 
Only VOD extractor has been implemented

Related: ytdl-org/youtube-dl#26125
Related: blackjack4494#220
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merval merval merval left review comments

@blackjack4494 blackjack4494 blackjack4494 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@GreyAlien502 @blackjack4494 @merval @shiwei87tan @tanliang2 @pukkandan